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The  benefits  of  completing  homework  for  students  with  different  aptitudes  in  an  introductory 

electricity  and  magnetism  course 

F ,  J,  Kontur,  K.  de  La  Harpe,  and  N.  B.  Terry 

Department  of  Physics,  United  States  Air  Force  Academy,  USAF  Academy,  Colorado  80840,  USA 

We  examine  how  student  aptitudes  impact  how  much  students  learn  from  doing  graded  online 
and  written  homework  in  an  introductory  electricity  and  magnetism  course.  Our  analysis  examines  the 
correlation  between  successful  homework  completion  rates  and  exam  performance  as  well  as  how 
changes  in  homework  completion  correlate  with  changes  in  exam  scores  for  students  with  different 
physics  aptitudes.  On  average,  successfully  completing  many  homework  problems  correlated  to  better 
exam  scores  only  for  students  with  high  physics  aptitude.  On  the  other  hand,  all  other  students  showed 
zero  or  even  a  negative  correlation  between  successful  homework  completion  and  exam  performance. 
Low-  and  medium  aptitude  students  who  did  more  homework  did  no  better  and  sometimes  scored  lower 
on  exams  than  their  low-  and  medium-aptitude  peers  who  did  less  homework.  Our  work  also  shows  that 
long-term  changes  in  homework  completion  correlated  to  long  term  changes  in  exam  scores  only  for 
students  with  high  physics  aptitude,  but  not  for  students  with  medium-  or  low-aptitude.  We  offer  several 
explanations  for  the  disparity  in  homework  learning  gains,  including  cognitive  toad  theory,  ineffective 
homework  strategies,  and  various  mismatches  between  homework  and  exams*  Several  solutions  are 
proposed  to  address  these  possible  deficiencies  in  graded  online  and  written  homework. 


I.  INTRODUCTION 

Homework  is  a  key  part  of  nearly  every  college-level  physics  course  principally  because  both 
physics  professors  and  students  believe  to  be  an  effective  tool  for  learning  physics.  That  is,  doing  more 
homework  will  lead  to  greater  understanding  of  physics  concepts  and  increased  success  on  exams. 

Because  of  its  prevalent  use  in  introductory  physics  courses,  it  is  not  surprising  that  homework  has  been 
one  of  the  most  well-studied  aspects  of  physics  pedagogy.  Numerous  articles  examined  The  advantages 
and  disadvantages  of  online  homework  [1,2,  3,  4,  5];  other  research  on  homework  studied  student 
motivation  [6],  the  deficiency  of  traditional  homework  in  teaching  physics  concepts  [7],  and  ways  to  deal 
with  homework  copying  [8,  9,  10],  Despite  the  large  amount  of  research  that  has  been  done  on  homework 
in  physics  courses,  many  questions  remain  about  how  homework  can  best  be  used  to  aid  student  learning, 
including  how  the  benefits  of  homework  depend  on  student  aptitude. 

While  at  least  three  articles  [11,12,  13]  touch  on  this  subject,  this  is  the  first  study  which  directly 
examines  the  question.  Cheng  et  ai  found  that  interactive  classes  combined  with  graded  online 
homework  led  to  the  higher  learning  gains  for  all  students  [11],  Similarly,  Kortemeyer  et  ai  found  that 
switching  to  an  online  homework  system  increased  course  grades  [12],  However,  these  two  studies  did 
not  directly  consider  the  impact  of  student  aptitude  on  the  effectiveness  of  homework.  Morote  and 
Prichard  found  that  academic  background  correlated  with  exam  scores  but  not  with  homework  scores 
[13],  and  while  their  results  suggest  that  homework  and  exam  scores  are  not  correlated,  their  work  did  not 
directly  exam  the  issue. 
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The  results  presented  here  suggest  that  incoming  student  aptitude  is  a  crucial  factor  determining 
how  much  students  learn  from  doing  homework.  We  characterized  the  impact  of  student  aptitude  by 
examining  how  both  successful  homework  completion  rales  and  changes  in  successful  homework 
completion  rales  correlate  to  exam  performance  for  students  with  different  physics  aptitude.  By 
characterizing  homework  effectiveness  in  these  two  ways,  we  find  that,  while  students  with  high 
incoming  physics  aptitude  get  some  benefit  on  course  exams  from  completing  homework,  students  with 
medium  and  low  incoming  aptitude  get  no  benefit  or  even  a  negative  benefit  on  course  exams  from 
completing  homework.  We  examine  several  possible  reasons  that  homework  yields  different  benefits  for 
students  with  different  physics  aptitudes.  Finally,  we  suggest  several  changes  to  make  homework  a  more 
valuable  learning  tool  for  a  wider  range  of  students.  As  societal  needs  and  public  policy  push  a  larger 
number  of  students  into  pursuing  science  and  technology  careers,  it  is  important  for  physics  educators  to 
know  how  beneficial  their  pedagogical  tools  are  for  students  having  a  wide  range  of  abilities.  We  hope 
that  this  investigation  will  be  helpful  for  teachers  dealing  with  such  issues  in  their  classrooms 

II.  METHODS 

We  begin  with  a  description  of  die  course  demographics,  content  and  homework  assignments  used 
during  the  semesters  studied  in  our  research.  All  students  at  the  United  States  Air  Force  Academy 
(USAFA)  take  two  semesters  of  calculus-based  introductory  physics,  although  only  about  half  of  these 
students  major  in  a  science,  technical,  engineering,  or  mathematics  field.  Students  generally  take 
Mechanics  in  their  freshman  year  and  Electricity  and  Magnetism  (E&M)  in  the  fall  semester  of  their 
sophomore  year,  unless  they  had  trouble  with  the  prerequisite  courses,  in  which  case  they  typically  take 
E&M  in  the  spring  semester.  Because  of  this,  the  fall  semester  of  E&M  generally  has  2-3  times  greater 
enrollment  than  the  spring  semester,  in  this  study,  we  chose  to  focus  on  the  E&M  course  rather  than  the 
Mechanics  course  for  several  reasons  -  students  have  had  less  high-school  exposure  to  the  material,  it  is 
more  challenging  than  the  Mechanics  course,  and,  being  later  in  the  course  sequence,  we  have  more  grade 
data  from  prerequisite  courses  to  characterize  student  aptitude. 

The  E&M  course  is  taught  in  sections  where  the  enrollment  is  set  at  approximately  20  students, 
with  between  5-12  different  instructors  teaching  the  course  each  semester.  All  sections  of  l he  course  use 
the  same  textbook,  use  the  same  syllabus,  complete  the  same  assignments,  and  take  the  same  quizzes  and 
exams.  During  the  semesters  considered  in  this  study,  the  E&M  course  textbook  was  Essential  University 
Physics  by  Richard  Wolfson  [14,  15].  All  USAFA  physics  instructors  are  trained  to  use  a  variety'  of 
interactive  teaching  techniques,  including  just-in-time  teaching  (JiTT)  [16],  peer  instruction  [17],  think- 
pair-share  [18],  and  board  work  problem-solving. 

Homework,  consisting  of  2-3  book  problems  for  each  lesson,  was  initially  administered  written, 
and  in  recent  semesters  online  through  the  Mastering  Physics  online  system.  Students  were  given  up  to  5 
tries  to  get  the  correct  answer  on  homework  problems,  with  no  deduction  for  an  incorrect  answer  until  the 
final  attempt.  The  amount  of  credit  given  for  homework  varied  from  a  low  of  6,0%  to  a  high  of  9.6%  of 
the  total  course  grade.  Homework  was  generally  due  each  lesson  or  each  week  to  discourage  cramming. 
For  the  purposes  of  this  study,  when  considering  online  homework,  we  used  the  Mastering  Physics 
homework  correctness  score  as  the  successful  homework  completion  score 

There  were  three  midterms  and  a  cumulative  final  exam  for  all  but  one  of  the  semesters  of  E&M 
studied.  The  midterms  were  each  worth  -10%  of  the  course  grade  and  the  final  exam  was  worth  25%.  In 
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the  spring  2009  E&M  course,  there  were  only  two  midterms,  which  were  each  worth  13.5%  of  the  course 
grade;  the  final  exam  was  worth  25%.  The  midterms  were  80-  to  1 10-minute  in-class  exams  comprised  of 
ten  conceptual  multiple-choice  questions  worth  50%  of  the  exam  points  and  two  or  three  homework -type 
workout  problems  worth  40-50%  of  the  exam  points  (8  of  the  20  midterms  also  had  a  short -answer 
question  that  was  worth  1 0%  of  the  exam  points).  The  final  exam  consisted  of  30-35  conceptual  multiple- 
choice  questions,  worth  60-70%  of  the  exam  points,  and  two  or  three  workout  problems,  worth  23-36%  of 
the  exam  points.  The  final  exam  sometimes  had  a  short  answer  question  as  well,  worth  between  6-17%  of 
the  exam  points.  In  addition  to  exams  and  homework,  the  remainder  of  the  course  points  was  a  mix  of 
grades  on  student  journals,  JiTT  assignments,  and  short  in-class  quizzes. 

The  first  drafts  of  the  exams  were  written  by  personnel  at  USAFA's  Center  for  Physics  Education 
Research,  These  personnel  were  not  instructors  in  the  E&M  course,  and  they  wrote  the  exams  using  the 
course  learning  objectives.  The  exams  were  edited  based  on  feedback  from  instructors.  In  general,  the 
editing  process  involved  re -wording  questions  to  make  them  more  understandable  and  replacement  of 
questions  based  on  which  concepts  were  emphasized  or  not  emphasized  in  class.  The  overall  goal 
throughout  the  exam  writing  and  editing  process  was  to  test  physics  understanding  rather  than 
memorization  or  pattern-matching. 

In  this  study,  course  exams  were  used  to  measure  student  learning  rather  than  research-based 
concept  tests  like  the  Force  Concept  Inventory  [19],  the  Conceptual  Survey  of  Electricity  and  Magnetism 
(CSEM)  [20]  or  the  Brief  Electricity  and  Magnetism  Assessment  [21].  The  first  reason  for  using  course 
exams  is  that  light  and  optics  (a  quarter  of  the  content  in  our  E&M  course)  is  not  covered  on  the 
aforementioned  concept  tests.  The  second  reason  is  that  the  concept  tests  assess  conceptual  knowledge 
rather  than  problem-solving  ability.  While  it  has  been  shown  that  increased  conceptual  knowledge  leads 
to  better  problem-solving  [17],  we  wanted  to  separately  identify  if  homework  was  benefitting  students’ 
conceptual  and/or  problem-solving  abilities.  In  this  study,  a  group  of  students  is  said  to  benefit  from 
doing  homework  if  there  was  a  positive  correlation  between  homework  scores  and  exam  scores. 

We  determined  physics  aptitude  by  identifying  the  variable  that  at  the  beginning  of  the  semester 
of  the  E&M  course,  best  predicted  student  success  on  E&M  exams.  While  there  were  positive  correlations 
between  grades  and  SAT  math  scores,  ACT  math  and  sciences  scores.  FCl  pre-  and  post-test  scores  and 
CSEM  pre-test  scores  (r3=0. 1 4-0.3 1)  the  strongest  predictor  of  exam  scores  were  students’  combined 
grade  point  averages  (GPA)  in  three  incoming  prerequisite  courses  (Calculus  1  and  2  and  Mechanics  ), 
with  r=0.52.  The  high  correlation  between  calculus  grades  and  E&M  exam  performance  is  consistent 
with  previous  studies  that  found  a  high  correlation  between  students'  math  skills  and  their  exam  grades  in 
college  physics  [22]. 

Students’  combined  GPA  in  Calculus  1,  Calculus  2,  and  Mechanics  was  used  to  define  student’s 
aptitudes.  The  highest  GPA  that  can  be  achieved  is  4.0,  for  a  student  who  received  an  A  in  all  three 
courses.  The  lowest  GPA  is  1 .0,  for  a  student  who  received  a  D  in  all  three  courses.  A  student  who  fails 
one  of  the  courses  must  retake  it  before  taking  E&M.  For  simplicity,  in  the  case  of  students  who  took  a 
course  more  than  once,  we  only  considered  their  most  recent  grade  in  the  course.  As  shown  in  Table  I,  we 
grouped  students  into  four  physics  aptitude  groups  based  on  their  GP As  in  the  three  pre-requisite  courses. 
Each  aptitude  group  covers  0.75  grade  points,  from  the  maximum  of  4  to  the  minimum  of  l .  For  the 
remainder  of  this  article,  the  aptitude  groups  will  be  referred  to  using  their  names  in  Table  1. 
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TABLE  1.  We  divided  students  into  four  different  physics  aptitude  groups  according  to  their  GPA  in  Calculus  l. 
Calculus  2,  and  Mechanics. 


GPA 

Physics  Aptitude  Group 

4.00-3.25 

High 

3.24-2.50 

Medium-High 

2,49-1.75 

Medium-Low 

1.74-  1.00 

Low 

HI.  RESULTS 

In  our  analysis,  we  examine  the  correlation  between  successful  homework  completion  and  exam 
scores  for  each  aptitude  group  described  in  previous  section.  We  also  consider  the  correlation  between 
long  term  changes  in  successful  homework  completion  rates  and  long  term  changes  in  exam  performance 
for  each  aptitude  group.  Before  we  continue,  it  is  worth  noting  that  we  as  instructors  assign  homework  to 
our  students  because  we  believe  that  all  students  will  receive  a  learning  benefit  from  doing  homework. 
This  belief  is  predicated  on  the  idea  that  there  is  a  strong  positive  correlation  between  making  an  honest 
effort  to  do  well-chosen  homework  problems  and  student  learning,  qualities  we  attempt  to  measure  using 
homework  scores  and  exam  scores,  If  there  is  no  correlation  between  successful  homework  completion 
and  exam  scores,  we  should  carefully  rethink  these  assumptions,  especially  considering  that  assigning, 
completing,  and  grading  homework  is  often  a  very  time- intensive  activity  for  both  students  and  teachers* 
The  remainder  of  this  section  is  devoted  to  examining  the  strength  of  the  expected  correlation  between 
homework  scores  and  learning  for  all  of  our  students. 

Fig.  1  plots  the  total  exam  score  (the  average,  weighted  by  percentage  of  overall  course  grade,  of 
the  scores  on  the  midterm  exams  plus  the  filial  exam)  versus  homework  completion  score  (as  mentioned 
in  Section  If  this  is  the  same  as  the  Mastering  Physics  correctness  score)  for  each  student  who  took  E&M 
in  the  fall  2010  and  spring  201 1  semesters.  Included  with  the  plots  are  line  fits  for  the  data.  As  mentioned 
in  Section  II,  the  fall  semesters  of  E&M  have  2-3  times  greater  enrollment  than  the  spring  semesters,  so 
there  is  a  larger  set  of  data  for  the  fall  2010  semester  compared  to  the  spring  201 1  semester.  Remarkably, 
medium-low  and  low  aptitude  students  have  a  negative  benefit  from  doing  homework  for  both  semesters 
shown  in  Fig.  L  This  indicates  that,  on  average,  the  more  homework  that  medium-low  and  low  aptitude 
students  did,  the  worse  they  performed  on  exams. 

A  visual  inspection  of  Fig.  1  might  suggest  that  students  completing  less  than  50%  of  the 
homework  are  the  dominant  influence  on  the  slope  of  the  graphs  and  that  removing  them  from  the 
analysis  would  make  the  slopes  positive,  especially  for  the  low-  and  medium-low  aptitude  students. 

1  lowever,  removing  the  6%  of  students  who  successfully  completed  less  than  50%  of  the  homework  from 
the  analysis  only  changes  the  slope  of  the  graph  for  the  low-  and  medium-low  aptitude  students  by  about 
1%.  The  slopes  are  still  slightly  negative,  and  certainly  not  strongly  positive  as  predicted  by  our 
commons  sense  notions  of  homework.  On  the  other  hand,  removing  the  6-12%  of  students  who 
successfully  completed  less  than  50%  of  the  homework  from  the  analysis  increases  the  slope  of  the  graph 
by  about  4%  for  the  medium-high  and  the  high  aptitude  students*  This  is  at  least  consistent  with  (or 
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FIG.  1,  The  weighted-average  score  on  all  exams  vs.  homework  completion  score  for  students  in  different  aptitude 
groups  who  look  the  USA  FA  E&M  course  in  fall  2010  (left  plots)  and  spring  201 1  (right  plots).  Line  fits  are 
included  with  the  data.  The  semesters  shown  here  are  representative  of  the  correlations  of  the  other  7  semesters 
studied. 
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perhaps  even  strengthens)  our  conclusion  that  homework  appears  to  he  “working”  For  our  best  students, 
but  not  for  our  struggling  students.  The  negative  benefit  effect  for  medium-low  and  low  aptitude  students 
occurred  in  6  of  the  7  semesters  analyzed  in  this  study. 

Fig.  2  summarizes  the  results  of  Fig.  1  by  illustrating  the  average  benefits  of  successful 
homework  completion  real  ized  by  students  of  various  aptitudes  during  the  7  semesters  we  studied.  Each 
semester  of  the  7  semesters  is  first  analyzed  independently  and  then  combined  (rather  than  simply 
grouping  all  the  semesters  together  for  a  single  analysis)  to  control  for  variability  between  exams  from 
semester  to  semester.  This  way  the  learning  activity  (homework)  for  each  semester  is  only  compared 
against  the  learning  measurement  (exams)  for  that  semester.  The  error  bars  show  the  standard  deviation  of 
the  slopes  from  the  7  semesters.  Fig.  2  shows  that,  on  average,  medium-low  and  low  aptitude  students  had 
a  significant*  consistent  and  repeatable  negative  benefit  from  completing  homework.  The  results  in  Fig.  2 
are  almost  identical  if  the  conceptual  multiple  choice  question  portions  of  the  exam  and  the  homework- 
style  workout  problems  on  the  exam  are  considered  separately. 
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FIG.  2,  Average  of  the  slopes  of  the  l  ine  fils  between  successful  homework  completion  and  exam  score  For  foul- 
different  aptitude  groups.  The  data  is  from  7  semesters  of  E&M  when  homework  was  administered  online  and 
graded  only  For  correctness.  Error  bars  are  the  standard  deviations  of  the  slopes  for  the  7  different  semesters. 


Prior  to  spring  2009,  the  USAFA  E&M  courses  often  employed  a  combination  of  instructor- 
graded  written  homework  and  computer-graded  Mastering  Physics  homework.  In  some  semesters,  the 
written  homework  would  be  students'  solutions  to  original  problems  written  by  the  lead  teacher  for  the 
course,  and,  in  other  semesters,  the  written  homework  would  be  students*  written  solutions  to  the 
Mastering  Physics  problems.  Regardless,  homework  data  from  those  semesters  should  indicate  if  written 
homework  is  more  effective  than  online  homework  for  lower  aptitude  students.  The  specific  semesters  we 
examined  were  the  fall  2006  -  spring  2008  semesters  of  E&M.  Similar  to  the  analysis  in  Fig.  2,  we 
graphed  written  homework  score  versus  total  exam  score  for  different  aptitude  students  and  determined 
the  slopes  of  those  plots.  Taking  the  average  slopes  from  those  four  semesters  yields  the  data  shown  in 
Fig.  3.  The  data  show  that,  in  fact,  the  negative  benefits  seen  in  Fig.  2  are  no  longer  present  for  the  written 
homework.  However,  it  appears  that  written  homework  is  only  effective  for  high  aptitude  students.  For  all 
other  students,  homework  has  essentially  no  effect  on  total  exam  scores.  We  assign  homework  because 
we  expect  there  is  a  causative  relationship  between  homework  and  learning.  The  correlations  in  Fig.  2 
and  Fig.  3  suggest  that  this  anticipated  causative  relationship  should  be  reevaluated  for  major  segments  of 
our  student  population. 
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FIG.  3,  Average  of  l  he  slopes  of  written  homework  scores  vs.  total  exam  score  plots  as  a  function  of  student 
aptitude  for  four  semesters  when  homework  problems  were  administered  in  a  graded,  written  format.  Error  bars  are 
the  standard  deviations  of  the  slopes  for  the  4  different  semesters  where  written  homework  was  used. 


Besides  determining  the  correlation  between  total  homework  completion  and  test  scores,  we  also 
examined  a  second  common  sense  prediction  that  students  who  change  their  homework  behavior  by 
increasing  the  amount  of  horpework  they  do  will  generally  benefit.  To  test  this  prediction,  we  examined 
the  changes  in  homework  completion  rates  of  students  who  took  Mechanics  and  E&M  in  successive 
semesters.  For  each  student  in  E&Mt  we  measured  the  change  in  successfully  completed  homework 
between  the  two  courses  and  compared  this  to  the  change  in  total  exam  scores  for  each  student.  We  were 
only  able  to  examine  5  of  the  7  semesters  because  it  was  not  possible  to  determine  the  change  in 
homework  score  for  two  semesters  where  homework  was  ungraded  in  the  Mechanics  course.  Fig.  4 
shows  the  change  in  exam  scores  as  a  function  of  the  change  in  homework  scores  for  students  who  took 
Mechanics  in  spring  2009  and  E&M  in  fall  2009.  The  average  slopes  of  the  changes  in  homework  vs 
change  in  homework  for  all  5  semesters  are  shown  in  Fig.  5,  Again,  we  see  that  high  aptitude  students 
show  a  consistent  benefit  from  doing  more  homework  but  other  students  do  not. 

While  Fig.  4  and  5  examine  changes  in  student  behavior  changes  between  mechanics  and  E&M, 
we  also  examined  changes  in  student  behavior  within  the  E&M  course.  Fig.  6  looks  at  the  change  in 
individual  exam  scores,  from  one  exam  to  the  next,  as  a  function  of  the  change  in  homework  score,  from 
one  exam  to  the  next  within  a  semester  of  E&M,  The  graph  labeled  Midterm  2  plots  the  change  in  exam 
score  from  Midterm  I  to  Midterm  2  as  a  function  of  the  change  in  homework  score  from  course  block  l  to 
course  block  2.  Similarly,  the  graph  labeled  Midterm  3  plots  the  change  in  exam  score  from  the  average 
of  Midterms  1  and  2  to  Midterm  3  as  a  function  of  the  change  in  the  average  homework  score  from  course 
blocks  1  and  2  to  course  block  3, 

Fig.  5  shows  that  increasing  homework  completion  from  Mechanics  to  E&M  only  benefits  high 
aptitude  students  on  exams,  which  is  consistent  with  the  other  data  presented  thus  far.  However,  though 
the  data  are  noisy.  Fig,  6  shows  that  increasing  homework  completion  before  an  exam  results  in  better 
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FIG.  4.  Plots  of  the  change  in  the  total  exam  score  versus  the  change  in  homework  score  for  students  moving  from 
spring  2009  Mechanics  to  fall  2009  E&M.  Each  plot  shows  students  in  different  aptitude  groups.  Line  fits  are 
included  with  the  data.  These  results  shown  here  are  typical  of  all  the  semesters  we  examined. 
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FIG.  5.  The  average  correlation  between  the  change  in  HW  score  versus  change  in  total  exam  score  from 
Mechanics  to  E&.M  as  a  function  of  student  aptitude  for  5  semesters  of  E&M  being  studied.  Error  bars  are  the 
standard  deviations  of  the  slopes  for  the  different  semesters. 
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FIG,  6,  Average  slopes  ofline  fit  data  of  change  in  homework  score  vs.  change  in  exam  score  (left)  from  Midterm 
I  to  Midterm  2,  and  (right)  from  Midterms  J  and  2  to  Midterm  3.  Only  6  of  the  7  semesters  being  studied  are 
included  in  this  analysis,  with  spring  2009  left  out  because  there  were  only  2  midterms  during  that  semester 


performance,  on  average,  on  that  exam  for  nearly  all  students.  In  particular,  medium-low  aptitude 
students  clearly  show  positive  benefits  from  increasing  their  rate  of  homework  completion  in  the  lessons 
leading  to  an  exam.  This  is  likely  why  students  have  such  a  strong  belief  that  homework  is  beneficial. 
They  can  see  immediate  benefits,  on  the  next  midterm,  when  they  do  more  homework.  Unfortunately, 

Fig.  5  shows  that  a  longterm  increase  in  homework  completion  does  not  correspond  to  a  long  term 
increase  in  learning  for  low-  or  medium -aptitude  students.  Instead,  the  benefits  due  to  increasing 
homework  effort  appear  to  only  yield  short  term  rather  than  long  term  benefits  for  all  but  our  best 
students, 

IV,  DISCUSSION 

While  instructors  assign  homework  because  we  believe  all  our  students  will  learn  from  doing  it, 
the  data  presented  in  the  previous  sections  contradicts  this  belief,  instead  suggesting  that  homework  is  an 
effective  learning  tool  only  for  our  best  students,  it  is  important  to  note  that  correlation  does  not  imply 
causation,  meaning  that  the  data  does  not  tell  us  why  homework  doesn't  work  as  expected.  If  our  low- 
aptitude  students  are  getting  something  out  of  homework*  it  is  clearly  not  what  we  expect  them  to  learn. 
There  are  a  number  of  possible  explanations  for  why  homework  does  not  work  for  our  struggling 
students,  ranging  from  how  students  approach  problems,  what  problems  we  assign,  the  sequencing  of 
homework  problems,  the  ways  which  we  use  to  motivate  homework,  how  lessons  couple  with  follow-up 
homework,  and  how  exams  measure  what  is  learned  by  doing  homework,  to  name  a  few.  Fully 
understanding  the  problem  of  why  homework  is  only  effective  for  our  high-aptitude  students  is  an  area 
where  significant  productive  research  remains  to  be  done. 

We  begin  this  discussion  by  considering  three  possible  reasons  that  homework  is  ineffective: 
students  are  not  working  the  homework  problems  in  an  effective  manner,  our  exams  are  not  appropriately 
measure  student  learning,  or  we  are  not  assigning  the  right  problems.  However,  we  are  not  proposing  any 
of  these  ideas  as  a  causative  explanation  based  on  the  data  presented  in  this  article.  Our  major  finding  is 
that,  even  though  most  teachers  do  think  there  is  a  connection  between  doing  more  homework  and  doing 
better  on  exams,  our  analysis  finds  that  no  such  connection  exists  for  most  students. 
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A.  Homework  Copying 

We  begin  by  considering  the  prevalence  of  copying  [8,  9],  which  is  often  an  attempt  to  save  time 
and  effort  and  can  range  from  blatant  cheating  to  a  misguided  overreliance  on  solution  manuals.  To 
characterize  the  levels  of  homework  copying  in  our  course,  we  went  through  19  different  problem  sets  of 
Mastering  Physics  for  the  fall  2010  semester  and  measured  the  number  of  attempts  used  by  randomly 
selected  low  and  high  aptitude  students.  We  then  determined  the  ease  with  which  these  students  did  the 
homework  by  dividing  their  homework  score  by  the  number  of  attempts  they  used.  We  excluded  students 
who  did  less  than  50%  of  the  homework  so  they  would  not  skew  the  data.  There  was  no  statistically 
significant  correlation  between  the  number  of  points  students  earned  per  attempt  and  their  overall  exam 
score.  We  also  found  no  correlation  between  test  scores  and  the  amount  of  time  students  were  logged  into 
the  Mastering  Physics  homework  server.  Finally,  we  note  that  anonymous  surveys  of  student  homework 
behavior  showed  that  less  than  6%  of  students  self-reported  using  on-line  solutions  to  complete 
homework.  These  findings  suggest  that  copying  does  not  explain  why  homework  is  ineffective  for  our 
struggling  students. 

B.  Student  Approaches  to  Doing  Homework 

Having  ruled  out  copying  as  an  explanation  for  why  homework  is  ineffective  for  some  groups  of 
students,  but  not  others,  we  turn  our  attention  to  novice  approaches  to  homework.  For  many  students,  the 
key  to  doing  homework  problems  is  finding  the  right  equation.  Once  the  right  equation  is  found,  all  that  is 
necessary  is  to  plug-and-chug  to  get  the  correct  answer.  Unfortunately,  while  this  method  of  doing 
homework  may  teach  students  how  to  do  that  specific  problem,  it  does  not  teach  them  what  concept  or 
concepts  underlie  the  problem  and  how  the  concepts  in  the  problem  might  connect  to  concepts  that  they 
have  learned  previously  (see  p.  38  of  Ref.  18  for  a  visual  representation  of  this  phenomenon).  As  Redish 
at  aL  say,  “We  are  frustrated  by  the  tendency  many  students  have  to...  spend  a  large  amount  of  time 
memorizing  long  lists  of  un interpreted  facts  or  performing  algorithmic  solutions  to  large  numbers  of 
problems  without  giving  any  thought  or  trying  to  make  sense  of  them**  [22]. 

Unfortunately,  simply  telling  students  not  to  use  ineffective  strategies  probably  won't  solve  the 
problem.  Instead,  we  as  instructors  may  need  to  teach  them  effective  homework  strategies  to  replace  the 
ineffective  strategies.  The  first  step  to  doing  this  is  to  identify  effective  strategies.  This  is  more 
challenging  than  it  may  seem,  because  our  data  suggests  that  learning  strategies  which  are  effective  for 
high -aptitude  students  may  not  be  equally  effective  for  medium-  and  low -aptitude  students.  Different 
strategies  may  be  useful  for  these  latter  two  groups.  Identifying  and  characterizing  the  strategies  favored 
by  and  useful  for  each  aptitude  group  is  the  subject  of  a  future  article. 

C.  Validity  of  Course  Exams  in  Assessing  Learning 

Just  as  it  is  important  to  consider  the  strategies  our  students  employ  to  complete  their  homework, 
it  is  also  important  to  consider  how  we  measure  the  learning  gains  for  our  various  students.  Our  exams 
are  reviewed  by  multiple  physics  instructors  both  affiliated  and  not  affiliated  with  the  course.  One  of  the 
principal  things  that  is  considered  in  reviewing  the  exams  is  whether  it  is  properly  assessing  the  course 
objectives,  which  are  also  the  basts  for  the  homework  and  other  learning  exercises  m  the  course.  Our 
exams  consist  of  several  problems  for  students  to  solve,  some  of  which  students  have  seen  before  and 
some  which  are  subtly  different  than  they  have  seen  before.  We  also  administer  a  series  of  multiple- 
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choice  questions  that  probe  student  understanding  of  electricity  and  magnetism  concepts.  From  this 
perspective,  our  exams  have  face  validity  and  are  probing  the  concepts  and  skills  we  are  trying  to  teach 
our  students. 

On  the  other  hand,  it  is  quite  possible  students  with  low  incoming  grades  in  prerequisite  courses 
are  learning  something  from  homework,  but  what  they  are  learning  are  basic  math  and  science  skills, 
rather  than  the  higher  level  physics  conceptual  and  problem-solving  skills  probed  by  our  exams. 

[  lowever,  their  high-aptitude  peers  have  already  learned  the  basic  skills  and  instead  can  focus  on  learning 
advanced  concepts  and  problem-solving  skills  from  doing  homework.  While  successful  completion  of 
exam  problems  demands  a  mastery  of  algebra  (for  example),  our  exams  are  not  designed  to  accurately 
differentiate  between  levels  of  algebra  mastery.  Perhaps  the  optimum  balance  of  simple  homework 
problems  that  review  basic  skills  and  homework  problems  on  more  advanced  topics  is  different  for  high- 
and  low-aptitude  students.  For  example,  conceptual  questions  on  electric  forces  that  are  designed  to  probe 
students’  understanding  of  electric  fields  also  require  a  basic  understanding  of  forces.  Perhaps  students 
completing  homework  problems  dealing  with  electromagnetic  forces  are  able  to  solidify  their 
understanding  of  the  basic  idea  of  Newtonian  forces.  Understanding  of  Newtonian  forces  is  not  well- 
differentiated  by  conceptual  questions  about  electromagnetic  forces,  meaning  a  student  is  penalized  for 
not  learning  about  electric  forces  and  remains  unrewarded  for  gains  in  their  understanding  of  Newtonian 
forces.  Our  exams  may  simply  not  measure  gains  in  fundamental  skills  which  we  assumed  (perhaps 
erroneously)  that  students  have  previously  mastered,  even  if  these  gains  in  fundamental  skills  are  very 
large. 

D.  Cognitive  Load 

Having  considered  effective  homework  strategies  and  what  our  exams  arc  actually  measuring,  the 
final  possibility  we  consider  is  that  there  is  something  intrinsically  inappropriate  with  the  homework 
problems  that  we  give  our  students  for  practice.  There  are  a  number  of  interconnected  parameters  which 
might  impact  the  effectiveness  of  homework,  including  number  of  assigned  problems,  sequencing, 
difficulty'  level,  student  motivation,  instructor  feedback,  grading  practices,  etc.  For  example,  while  exams 
require  students  to  solve  a  particular  problem  selected  from  a  group  of  topics,  homework  problems  are 
often  linked  to  a  particular  lesson  which  only  covers  one  or  two  physics  concepts.  While  exams  require 
students  to  remember  concepts  and  problem  solving  strategies,  homework  assignments  typically  allow 
students  to  use  their  textbook,  notes,  and  other  sources.  Since  it  would  be  difficult  to  consider  all  of  these 
variables  in  a  single  study,  we  instead  discuss  principles  which  explain  the  effectiveness  of  homework  as 
a  function  of  student  aptitude. 

One  principle  which  can  explain  differential  learning  gains  is  cognitive  load  theory',  which  posits 
that  learners  can  simultaneously  process  only  a  limited  number  of  ideas  in  their  working  memory  [23,  24 1. 
Additional  processing  power  is  available  from  long-term  memory,  provided  that  students  have  that 
knowledge  encoded  in  long-term  memory.  Know' ledge  thus  stored  in  long-term  memory  is  known  as  a 
schema.  High-aptitude  students  appear  to  possess  a  large  number  of  useful  schemas,  both  from  their 
previous  courses  and  from  an  ability  to  better  incorporate  early  course  material  into  new  schemas.  As 
homework  problems  become  more  complicated,  high-aptitude  students  who  call  upon  these  schemas  can 
supplement  their  limited  working  memory,  enabling  them  to  process  the  homework  and  use  it  to  learn 
new  material. 
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On  the  other  hand,  low-aptilude  students  possess  only  a  limited  number  of  schemas  from  previous 
courses  and  it  possible  that  some  of  these  schemas  may  even  be  incorrect.  Thus,  low-aptitude  students  can 
only  supplement  their  working  memory  with  a  very  small  number  of  correct  schemas.  In  some  cases, 
low-aptitude  students  may  build  schemas  associated  with  basic  skills  that  instructors  already  expect 
students  to  know.  On  the  other  hand,  low-aptitude  students  may  regularly  experience  cognitive  overload 
when  dealing  with  complicated  problems.  Students  may  deal  w  ith  this  cognitive  overload  by  creating  new 
schemas  for  each  homework  problem,  schema  which  may  focus  on  surface  features  of  the  problem  rather 
than  on  deep  physics. 

Inevitably,  the  inefficient  schema-building  that  occurs  for  low-aptitude  students  while  doing 
homework  problems  hinders  their  success  on  exams.  This  will  have  a  cumulative  effect,  because  a  proper 
understanding  of  physics  requires  students  to  build  a  coherent  framework  of  concepts  during  the  semester. 
A  student  who  has  separate,  often  incorrect,  schema  for  all  of  the  different  homework  problems  (based  on 
surface  features,  for  example)  will  have  an  increasingly  difficult  time  recognizing  the  conceptual 
framework  that  is  being  built  over  the  semester.  Therefore,  though  they  might  be  able  to  memorize  and/or 
pattern-match  enough  to  do  adequately  on  an  exam  that  tests  a  limited  number  of  concepts,  low-aptitude 
students  will  fail  to  achieve  the  big  picture  understanding  that  is  necessary  to  do  well  on  a  more 
comprehensive  test,  such  as  the  final  exam.  We  expect  that,  for  our  medium-aptitude  students, 
knowledge  in  long-term  memory  would  sufficiently  supplement  their  working  memories  to  prevent 
cognitive  overload  for  some  problems  and  not  others.  They  would  therefore  leant  from  some  homework 
problems  and  not  others,  making  their  learning  gains  somewhere  between  those  of  the  high -aptitude  and 
the  low-aptitude  students. 

E.  Possible  Solutions 

Before  we  suggest  solutions  to  the  aforementioned  difficulties,  we  note  both  cognitive  load  theory 
and  the  idea  that  struggling  students  only  learn  basic  skills  from  homework  both  explain  the  differential 
learning  gains  between  students  of  various  aptitudes.  These  ideas  are  interconnected.  We  highlight  this 
point  to  emphasize  that  student  preparation  is  the  main  problem  rather  than  specific  systems  of 
homework.  A  possible  solution  to  the  issue  of  homework  placing  an  excessive  cognitive  load  on 
low-aptitude  students  is  to  assign  easier  homework  problems  which  focus  on  fundamental  skills. 
Teoderescu  et  ai  attempted  this  with  pre-class  work  [25].  Their  results  indicate  "’that  choice  allows 
students  to  earn  a  reasonable  amount  of  points  with  problems  that  match  their  ability/’  However,  they  also 
found  that,  for  more  difficult  problems,  ”  C  and  D  [students]  may  simply  elect  not  to  invest  more  time 
when  their  success  rate  drops,  even  when  sufficient  points  have  not  been  accumulated.”  The  issue  with 
this  is  that  easier  homework  problems  may  leave  students  ill-prepared  for  exams  testing  advanced  skills. 
Another  option  is  strengthening  prerequisite  course  requirements  [26]. 

Another  possible  solution  that  has  met  with  some  success  in  previous  studies  is  to  require  mastery 
of  pre-requisite  knowledge  before  a  student  can  move  on  to  the  next  learning  topic.  This  would  require 
the  students  to  achieve  the  level  of  understanding  of  high-aptitude  students  before  moving  from 
Mechanics  to  E&M  or  before  moving  from  one  section  of  the  E&M  course  to  the  next.  One  study  of  an 
E&M  course  that  employed  mastery  learning  found  that  students  who  went  through  the  course  did  10- 
15%  better  on  the  llnal  exam  compared  to  equivalent  students  in  a  traditional  course  [27].  In  addition,  on 
a  re-test  of  the  course  material  given  two  months  later,  students  who  were  in  the  mastery  learning  course 
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did  1 5-20%  better  than  students  who  took  the  traditional  course,  indicating  that  not  only  did  the  students 
learn  more  in  the  mastery  learning  course,  they  retained  what  they  learned  for  a  longer  period  of  time.  In 
addition,  a  meta-analysis  of  mastery  learning  courses  found  that  the  average  improvement  of 
high-aptitude  students  in  such  courses  is  0.40  standard  deviations  while  the  average  improvement  of 
low-aptitude  students  is  0.61  standard  deviations  [28].  This  is  consistent  with  our  findings  that  while 
high-aptitude  students  benefit  from  doing  homework,  struggling  students  do  not  get  much  benefit  from 
homework.  If  low- aptitude  students  are  required  to  demonstrate  mastery  of  each  subject  before  moving 
on,  not  only  will  they  start  each  new  section  of  the  course  with  more  pre-requisite  knowledge,  they  should 
also  get  more  out  of  the  time  that  they  put  into  homework  and  quite  possibly  other  course  learning 
activities. 

One  final  suggestion  is  that  in  order  to  make  the  skills  practiced  in  homework  better  match  the 
skills  required  for  success  on  exams,  instructors  can  make  the  homework  more  like  exams.  In  a  review  of 
research  on  learning,  Rohrer  and  Pashler  found  that  "'a  combination  of  study  and  tests  is  snore  effective 
than  spending  the  same  amount  of  time  reviewing  the  material  in  some  other  way”  [29].  In  addition  to  the 
learning  advantages  of  testing  versus  reviewing,  Rohrer  and  Pashler  also  discuss  a  technique  called 
interleaving  [29].  In  interleaving,  learning  activities  are  not  grouped  together  by  subject.  Rather,  several 
different  subjects  are  mixed  together  in  the  learning  activities.  So,  for  example,  if  Faraday’s  law  is 
covered  in  a  particular  lesson,  the  homework  for  that  lesson  could  have  Coulomb’s  law  and  circuit 
analysis  and  magnetic  force  problems  in  addition  to  Faraday’s  law  problems.  The  authors  cite  one  study 
where  mathematics  test  scores  increased  by  a  factor  of  three  when  interleaving  was  employed  [30].  As 
discussed  above,  one  of  the  skills  required  for  success  on  exams  that  is  not  typically  practiced  on 
homework  is  identify  ing  which  concept  applies  to  a  particular  problem.  Interleaved  homework  remedies 
this  issue. 


V.  SUMMARY  AND  CONCLUSIONS 

This  paper  examined  the  effectiveness  of  online  and  written  homework  as  a  function  of  student 
aptitude.  We  studied  the  effectiveness  of  homework  by  determining  the  correlation  between  successful 
homework  completion  and  exams  scores  and  the  correlation  between  changes  in  homework  completion 
and  changes  in  exam  scores.  We  were  surprised  to  find  that  only  high -aptitude  students  seemed  to  derive 
any  measurable  benefit  from  completing  homework,  while  all  other  students  received  zero  or  even 
negative  benefit  from  doing  homework.  This  result  is  troubling  because  doing  homework  is  typically 
considered  a  key  to  success  in  physics  courses,  so  much  so  that  struggling  students  are  often  advised  to  do 
more  homework  in  order  to  be  better  prepared  for  exams.  Our  findings  indicate  that  this  learning  strategy 
may  be  completely  ineffective  for  most  of  our  students. 

We  explained  these  results  in  terms  of  student  preparation  and  argued  that  the  results  are  not  due 
to  a  particular  way  of  administering  homework  but  rather  arise  from  the  fact  that  poor  fundamental  skills 
in  mathematics  or  Newtonian  mechanics  prevent  students  from  learning  the  more  advanced  skills  which 
are  the  focus  of  an  introductory  electricity  and  magnetism  course.  Homework  imposes  an  excessive 
cognitive  load  on  low-aptitude  students  and  sometimes  even  medium-aptitude  students.  For  low-aptitude 
students,  this  may  encourage  a  focus  on  surface  features  of  homework  problems  rather  than  deeper 
learning.  We  also  discussed  how  homework  may  help  struggling  students  learn  basic  skills  rather  than 
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more  advanced  skills  and  how  improvements  in  basic  skills  may  not  be  measured  by  course  exams. 
Lastly,  we  suggested  several  solutions  to  address  these  deficiencies  in  homework,  including  employing 
mastery  learning  in  physics  courses  and  interleaving  multiple  concepts  in  homework  assignments. 
Several  of  these  suggestions  should  be  fairly  easy  to  include  in  most  introductory  physics  courses.  Based 
on  the  evidence  presented  in  this  article,  making  such  changes  to  homework  could  lead  to  significant 
learning  gains. 


Notes 
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